{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "28a0673e-96b5-43f2-8a8b-bd033bf851b0",
   "metadata": {},
   "source": [
    "## The Big Project begins!!\n",
    "\n",
    "# \"THE PRICE IS RIGHT\" Capstone Project\n",
    "\n",
    "This week - build a model that predicts how much something costs from a description, based on a scrape of Amazon data\n",
    "\n",
    "# Order of play\n",
    "\n",
    "DAY 1: Data Curation  \n",
    "DAY 2: Data Pre-processing  \n",
    "DAY 3: Evaluation, Baselines, Traditional ML  \n",
    "DAY 4: Deep Learning and LLMs  \n",
    "DAY 5: Fine-tuning a Frontier Model  \n",
    "\n",
    "## DAY 1: Data Curation\n",
    "\n",
    "Today we'll scrub our dataset and curate our data\n",
    "\n",
    "The dataset is here:  \n",
    "https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023\n",
    "\n",
    "And the folder with all the product datasets is here:  \n",
    "https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023/tree/main/raw/meta_categories"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc53e053",
   "metadata": {},
   "source": [
    "<table style=\"margin: 0; text-align: left;\">\n",
    "    <tr>\n",
    "        <td style=\"width: 150px; height: 150px; vertical-align: middle;\">\n",
    "            <img src=\"../assets/business.jpg\" width=\"150\" height=\"150\" style=\"display: block;\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <h2 style=\"color:#181;\">Business value of Data Curation</h2>\n",
    "            <span style=\"color:#181;\">Data Curation can be considered the less glamorous work of a Data Scientist. I say that's nonsense!\n",
    "            This is where the science happens - what could be more glamorous than that?! R&D with your\n",
    "            dataset can often have a greater impact on performance than the fashionable 'hyper-parameter optimization' that we do later.\n",
    "            So: prepare for Quality Time with Data Quality.</span>\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67cedf85-8125-4322-998e-9375fe745597",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from huggingface_hub import login\n",
    "from datasets import load_dataset\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm.notebook import tqdm\n",
    "import numpy as np\n",
    "import random\n",
    "from pricer.items import Item\n",
    "from pricer.parser import parse\n",
    "load_dotenv(override=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0732274a-aa6a-44fc-aee2-40dc8a8e4451",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log in to HuggingFace - if you get a \"Note\" about Environment variable being set, ignore it\n",
    "\n",
    "hf_token = os.environ['HF_TOKEN']\n",
    "login(hf_token, add_to_git_credential=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd6d801e-d195-45fe-898e-495dbcb19d7d",
   "metadata": {},
   "source": [
    "## Load our dataset\n",
    "\n",
    "In the next cell, we load in the dataset from huggingface.\n",
    "\n",
    "If this gives you an error like \"trust_remote_code is no longer supported\", then please run this command in a new cell: `!uv add --upgrade datasets==3.6.0` and then restart the Kernel, and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "049885d4-fdfa-4ff0-a932-4a2ed73928e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"McAuley-Lab/Amazon-Reviews-2023\", \"raw_meta_Appliances\", split=\"full\", trust_remote_code=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cde08860-b393-49b8-a620-06a8c0990a64",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of Appliances: {len(dataset):,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e29a5ab-ca61-41cc-9b33-22d374681b85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Investigate a particular datapoint\n",
    "\n",
    "dataset[6]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c3a74d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# What's the most expensive item?\n",
    "\n",
    "max_price = 0\n",
    "max_item = None\n",
    "\n",
    "for datapoint in tqdm(dataset):\n",
    "    try:\n",
    "        price = float(datapoint[\"price\"])\n",
    "        if price > max_price:\n",
    "            max_item = datapoint\n",
    "            max_price = price\n",
    "    except ValueError:\n",
    "        pass\n",
    "\n",
    "print(f\"The most expensive item is {max_item['title']} and it costs {max_price:,.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12d5934d",
   "metadata": {},
   "source": [
    "This is the closest I can find - looks like it's going at a bargain price!!\n",
    "\n",
    "https://www.amazon.com/TurboChef-Electric-Countertop-Microwave-Convection/dp/B01D05U9NO/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40a4e10f-6710-4780-a95e-6c0030c3fb87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load into Item objects if they have a price range $1-$1000 and enough details\n",
    "\n",
    "items = [parse(datapoint, \"Appliances\") for datapoint in tqdm(dataset)]\n",
    "items = [item for item in items if item is not None]\n",
    "print(f\"There are {len(items):,} items from {len(dataset):,} datapoints\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fa91b86",
   "metadata": {},
   "outputs": [],
   "source": [
    "items[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f9315f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(items[0].full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d356c6f-b6e8-4e01-98cd-c562d132aafa",
   "metadata": {},
   "outputs": [],
   "source": [
    "prices = [item.price for item in items]\n",
    "lengths = [len(item.full) for item in items]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89078cb1-9679-4eb0-b295-599b8586bcd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of lengths\n",
    "\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Lengths: Avg {sum(lengths)/len(lengths):,.0f} and highest {max(lengths):,}\\n\")\n",
    "plt.xlabel('Length (chars)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(lengths, rwidth=0.7, color=\"lightblue\", bins=range(0, 6000, 100))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "295be519",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_length = max(lengths)\n",
    "max_length_item = items[lengths.index(max_length)]\n",
    "print(max_length_item.full)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c38e0c43-9f7a-450e-a911-c94d37d9b9c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of prices\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Prices: Avg {sum(prices)/len(prices):,.2f} and highest {max(prices):,}\\n\")\n",
    "plt.xlabel('Price ($)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(prices, rwidth=0.7, color=\"orange\", bins=range(0, 1000, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3545170d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(items[3].full)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e60de0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pricer.loaders import ItemLoader\n",
    "loader = ItemLoader(\"Appliances\")\n",
    "items = loader.load()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4db93832",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "dataset_names = [\n",
    "    \"Automotive\",\n",
    "    \"Electronics\",\n",
    "    \"Office_Products\",\n",
    "    \"Tools_and_Home_Improvement\",\n",
    "    \"Cell_Phones_and_Accessories\",\n",
    "    \"Toys_and_Games\",\n",
    "    \"Appliances\",\n",
    "    \"Musical_Instruments\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6382c4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "items = []\n",
    "for dataset_name in dataset_names:\n",
    "    loader = ItemLoader(dataset_name)\n",
    "    items.extend(loader.load())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8225fd7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"A grand total of {len(items):,} items\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f5d7fd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "items[1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c25da1e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(42)\n",
    "random.shuffle(items)\n",
    "\n",
    "seen = set()\n",
    "items = [x for x in tqdm(items) if not (x.title in seen or seen.add(x.title))]\n",
    "\n",
    "seen = set()\n",
    "items = [x for x in tqdm(items) if not (x.full in seen or seen.add(x.full))]\n",
    "\n",
    "del seen\n",
    "print(f\"After deduplication, we have {len(items):,} items\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80d1cd91",
   "metadata": {},
   "outputs": [],
   "source": [
    "lengths = [len(item.full) for item in items]\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Text length: Avg {sum(lengths)/len(lengths):,.1f} and highest {max(lengths):,}\\n\")\n",
    "plt.xlabel('Length (characters)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(lengths, rwidth=0.7, color=\"skyblue\", bins=range(0, 4050, 50))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d8995c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of prices\n",
    "\n",
    "prices = [item.price for item in items]\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Prices: Avg {sum(prices)/len(prices):,.1f} and highest {max(prices):,}\\n\")\n",
    "plt.xlabel('Price ($)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(prices, rwidth=0.7, color=\"blueviolet\", bins=range(0, 1000, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e2955c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "category_counts = Counter([item.category for item in items])\n",
    "\n",
    "categories = category_counts.keys()\n",
    "counts = [category_counts[category] for category in categories]\n",
    "\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.bar(categories, counts, color=\"goldenrod\")\n",
    "plt.title('How many in each category')\n",
    "plt.xlabel('Categories')\n",
    "plt.ylabel('Count')\n",
    "plt.xticks(rotation=30, ha='right')\n",
    "\n",
    "for i, v in enumerate(counts):\n",
    "    plt.text(i, v, f\"{v:,}\", ha='center', va='bottom')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d70e9d40",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "SIZE = 820_000\n",
    "\n",
    "prices = np.array([it.price for it in items], dtype=float)\n",
    "categories = np.array([it.category for it in items])\n",
    "p = (prices - prices.min()) / (prices.max() - prices.min() + 1e-9)\n",
    "\n",
    "w = p**2\n",
    "w[categories == \"Tools_and_Home_Improvement\"] *= 0.5\n",
    "w[categories == \"Automotive\"] *= 0.05\n",
    "\n",
    "w = w / w.sum()\n",
    "idx = np.random.choice(len(items), size=SIZE, replace=False, p=w)\n",
    "sample = [items[i] for i in idx]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b695692b",
   "metadata": {},
   "outputs": [],
   "source": [
    "prices = [item.price for item in sample]\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Prices: Avg {sum(prices)/len(prices):,.1f} lowest {min(prices):,} and highest {max(prices):,}\\n\")\n",
    "plt.xlabel('Price ($)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(prices, rwidth=0.7, color=\"blueviolet\", bins=range(0, 1000, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1615044",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just for good measure, let's shuffle the sample again for the final dataset\n",
    "\n",
    "random.seed(42)\n",
    "random.shuffle(sample)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9bd3560",
   "metadata": {},
   "outputs": [],
   "source": [
    "prices = [item.price for item in sample]\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.title(f\"Prices: Avg {sum(prices)/len(prices):,.1f} lowest {min(prices):,} and highest {max(prices):,}\\n\")\n",
    "plt.xlabel('Price ($)')\n",
    "plt.ylabel('Count')\n",
    "plt.hist(prices, rwidth=0.7, color=\"blueviolet\", bins=range(0, 1000, 10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87fe08e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "category_counts = Counter([item.category for item in sample])\n",
    "\n",
    "categories = category_counts.keys()\n",
    "counts = [category_counts[category] for category in categories]\n",
    "\n",
    "# Bar chart by category\n",
    "plt.figure(figsize=(15, 6))\n",
    "plt.bar(categories, counts, color=\"goldenrod\")\n",
    "plt.title('How many in each category')\n",
    "plt.xlabel('Categories')\n",
    "plt.ylabel('Count')\n",
    "\n",
    "plt.xticks(rotation=30, ha='right')\n",
    "\n",
    "# Add value labels on top of each bar\n",
    "for i, v in enumerate(counts):\n",
    "    plt.text(i, v, f\"{v:,}\", ha='center', va='bottom')\n",
    "\n",
    "# Display the chart\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bcaa2d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Automotive still in the lead, but improved somewhat\n",
    "# For another perspective, let's look at a pie\n",
    "\n",
    "plt.figure(figsize=(12, 10))\n",
    "plt.pie(counts, labels=categories, autopct='%1.0f%%', startangle=90)\n",
    "\n",
    "# Add a circle at the center to create a donut chart (optional)\n",
    "centre_circle = plt.Circle((0,0), 0.70, fc='white')\n",
    "fig = plt.gcf()\n",
    "fig.gca().add_artist(centre_circle)\n",
    "plt.title('Categories')\n",
    "\n",
    "# Equal aspect ratio ensures that pie is drawn as a circle\n",
    "plt.axis('equal')  \n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7310efd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How does the price vary with the character count?\n",
    "\n",
    "sizes = [len(item.full) for item in sample]\n",
    "prices = [item.price for item in sample]\n",
    "\n",
    "# Create the scatter plot\n",
    "plt.figure(figsize=(15, 8))\n",
    "plt.scatter(sizes, prices, s=0.2, color=\"red\")\n",
    "\n",
    "# Add labels and title\n",
    "plt.xlabel('Size')\n",
    "plt.ylabel('Price')\n",
    "plt.title('Is there a simple correlation with text length?')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69c626cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How does the price vary with the weight?\n",
    "\n",
    "ounces = [item.weight for item in sample]\n",
    "prices = [item.price for item in sample]\n",
    "\n",
    "# Create the scatter plot\n",
    "plt.figure(figsize=(15, 8))\n",
    "plt.scatter(ounces, prices, s=0.2, color=\"darkorange\")\n",
    "\n",
    "# Add labels and title\n",
    "plt.xlabel('Weight (ounces)')\n",
    "plt.ylabel('Price')\n",
    "plt.xlim(0, 400)\n",
    "plt.title('Is there a simple correlation with weight?')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479d8fb2",
   "metadata": {},
   "source": [
    "## Now push this dataset to the HuggingFace Hub\n",
    "\n",
    "Replace the username with your HF username if you've crafted your own dataset\n",
    "\n",
    "Or, ignore this cell and you can load my dataset tomorrow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01e3ee1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "username = \"ed-donner\"\n",
    "full = f\"{username}/items_raw_full\"\n",
    "lite = f\"{username}/items_raw_lite\"\n",
    "\n",
    "train = sample[:800_000]\n",
    "val = sample[800_000:810_000]\n",
    "test = sample[810_000:]\n",
    "\n",
    "Item.push_to_hub(full, train, val, test)\n",
    "\n",
    "train_lite = train[:20_000]\n",
    "val_lite = val[:1_000]\n",
    "test_lite = test[:1_000]\n",
    "\n",
    "Item.push_to_hub(lite, train_lite, val_lite, test_lite)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b58dc61-747f-46f7-b9e0-c205db4f3e5e",
   "metadata": {},
   "source": [
    "## Sidenote\n",
    "\n",
    "If you like the variety of colors that matplotlib can use in its charts, you should bookmark this:\n",
    "\n",
    "https://matplotlib.org/stable/gallery/color/named_colors.html\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
